%-*-LaTeX-*-
\documentclass[11pt]{article}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Use the approved methods for setting lengths
%\oddsidemargin  0.0in
%\evensidemargin 0.0in
%\textwidth      6.5in
%\textheight     9.0in
\setlength{\oddsidemargin}{0.0in}
\setlength{\evensidemargin}{0.0in}
\setlength{\textwidth}{6.5in}
\setlength{\textheight}{8.7in}
% Based on document style, and taller text body height, set
% weird LaTex margin/header (box heights).  This fixes
% the disappearing page numbers (which never disappeared; they
% just got printed beyond the borders of the physical paper)
\setlength{\voffset}{0pt}
\setlength{\topmargin}{0pt}
\setlength{\headheight}{10pt}
\setlength{\headsep}{25pt}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\usepackage{color}
\usepackage{textpos}
%\usepackage{html}
\usepackage{makeidx}
%\usepackage{times}
\usepackage{hyperref}
\usepackage{verbatim}
\usepackage{graphicx}
%\usepackage{eufrak}
\usepackage{amsmath}

\makeindex

\tolerance=600

\newcommand{\Fb}{{\bf F}}
\newcommand{\Lb}{{\bf L}}
\newcommand{\gb}{{\bf g}}
\newcommand{\nb}{{\bf n}}
\newcommand{\ub}{{\bf u}}
\newcommand{\xb}{{\bf x}}
\newcommand{\p}{\partial}
\renewcommand{\div}{{\rm div}}
\renewcommand{\arraystretch}{1.3}

\begin{document}

\title{Installing SW4 version 2.0}

\author{ N. Anders Petersson$^*$ \and Bj\"orn Sj\"ogreen\thanks{Center for Applied Scientific
    Computing, Lawrence Livermore National Laboratory, PO Box 808, Livermore CA 94551. This work
    performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National
    Laboratory under contract DE-AC52-07NA27344. This is contribution
    LLNL-SM-741310.} }
\date{November 6, 2017}
\maketitle

%\thispagestyle{plain}

%\pagebreak


%\pagebreak
\tableofcontents

\section{Introduction}
The sole purpose of this document is to describe the installation process of the seismic wave
propagation code \emph{SW4}. A comprehensive user's guide is provided in the report by
Petersson and Sjogreen~\cite{SW4-v2}.

\section{Compilers and third party libraries}

Before you can build \emph{SW4} on your system, you must have
\begin{enumerate}
\item the \verb+lapack+ and \verb+blas+ libraries. These libraries provide basic linear algebra
  functionality and are pre-installed on many machines;
\item a MPI-2 library. This library provides support for message passing on parallel
  machines. Examples of open source implementations include Mpich-2 and OpenMPI. Note that the MPI-2
  library must be installed even if you are only building \emph{SW4} for a single core system.
\end{enumerate}
To avoid incompatibility issues and linking problems, we recommend using the same compiler for
the libraries as for \emph{SW4}.

In order to use geographic projection and material models stored in the \emph{rfile} format, you
need to install the Proj4 before building \emph{SW4}:
\begin{itemize}
\item The Proj.4 library, {\tt http://trac.osgeo.org/proj}
\end{itemize}  
If you also wish to use material models using the \emph{efile} format, you also need to download and
install two additional libraries:
\begin{itemize}
\item The Euclid e-tree library, {\tt http://www-2.cs.cmu.edu/~euclid}
\item The cencalvm library, \\{\tt http://earthquake.usgs.gov/data/3dgeologic/cencalvm\_doc/index.html}
\end{itemize}
To simplify the build process, all libraries should be installed under the same directory, such that
the library files (.so, .a, etc.) are in the lib sub-directory and the include files (.h) end up in
the include sub-directory. See Section~\ref{sec:cencalvm-install} for details.

\paragraph{Mac computers}

We recommend using the MacPorts package manager for installing the required compilers and
libraries. Simply go to www.macports.org, and install macports on your system. With that in place,
you can use the \verb+port+ command as follows
\begin{verbatim}
shell> sudo port install gcc72
shell> sudo port select --set gcc mp-gcc72
shell> sudo port install mpich-gcc7
shell> sudo port select --set mpi mpich-gcc7
\end{verbatim}
Here, \verb+gcc72+ refers to version 7.2 of the Gnu compiler suite. Compiler versions are bound to
change in the future, so the above commands will need to be modified accordingly. Before starting,
make sure you install a version of gcc that is compatible with the MPI library package. The above
example installs the \verb+mpich+ package using the gcc72 compilers, which includes a
compatible Fortran compiler. Alternatively, you can use the \verb+openmpi+ package. Note that the
\verb+port select+ commands are used to create shortcuts to the compilers and MPI environment. By
using the above setup, the Gnu compilers can be accessed with \verb+gcc+ and \verb+gfortran+
commands, and the MPI compilers and execution environment are called \verb+mpicxx+, \verb+mpif90+,
and \verb+mpirun+, respectively.

The \verb+lapack+ and \verb+blas+ libraries are preinstalled on recent Macs and can be accessed
using the \verb+-framework Accelerate+ link option. If that is not available or does not work on your
machine, you can download \verb+lapack+ and \verb+blas+ from www.netlib.org.

\paragraph{Linux machines}
We here give detailed instructions for installing the third part libraries under 64 bit, Fedora Core
18 Linux. Other Linux variants use similar commands for installing software packages, but note that 
the package manager \verb+yum+ is specific to Fedora Core.

You need to have root privileges to install precompiled packages. Start by opening an xterm and
set your user identity to root by the command
\begin{verbatim}
su -
\end{verbatim}
Install the compilers by issuing the commands
\begin{verbatim}
yum install gcc
yum install gcc-c++
yum install gcc-gfortran
\end{verbatim}
You install the \verb+mpich2+ library and include files with the command
\begin{verbatim}
yum install mpich2-devel
\end{verbatim}
The executables and libraries are installed in \verb+/usr/lib64/mpich2/bin+ and 
\verb+/usr/lib64/mpich2/lib+ respectively. We suggest that you add \verb+/usr/lib64/mpich2/bin+ to 
your path. This is done with the command
\begin{verbatim}
export PATH=${PATH}:/usr/lib64/mpich2/bin
\end{verbatim}
if your shell is \verb+bash+. For \verb+tcsh+ users, the command is
\begin{verbatim}
setenv PATH ${PATH}:/usr/lib64/mpich2/bin
\end{verbatim}
It is convenient to put the path setting command in
your startup file, \verb+.bashrc+ or \verb+.cshrc+., for bash or csh/tcsh respectively.

The \verb+blas+ and \verb+lapack+ libraries are installed with
\begin{verbatim}
yum install blas
yum install lapack
\end{verbatim}
On our system, the libraries were installed in \verb+/usr/lib64+ as \verb+libblas.so.3+ and
\verb+liblapack.so.3+. For some unknown reason, the install program does not add links to these
files with extension \verb+.so+, which is necessary for the linker to find them. We must therefore
add the links explicitly. If the libraries were installed elsewhere on your system, but you don't
know where, you can find them with the following command:
\begin{verbatim}
find / -name "*blas*" -print
\end{verbatim}
After locating the directory where the libraries reside (in this case \verb+/usr/lib64+), we add
links to the libraries with the commands:
\begin{verbatim}
cd /usr/lib64
ln -s libblas.so.3 libblas.so
ln -s liblapack.so.3 liblapack.so
\end{verbatim}
Note that you need to have \verb+root+ privileges for this to work.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Unpacking the source code tar ball}
\index{installation!directories}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

To unpack the \emph{SW4} source code, you place the file \verb+sw4-v2.0.tgz+ in the
desired directory and issue the following command:
\begin{verbatim}
shell> tar xzf sw4-v2.0.tgz
\end{verbatim}
As a result a new sub-directory named \verb+sw4-v2.0+ is created. It contains several files
and sub-directories:
%
\begin{itemize}
\item \verb+LICENSE.txt+ License information.
\item \verb+INSTALL.txt+ A link to this document.
\item \verb+README.txt+ General information about \emph{SW4}.
\item \verb+configs+ Directory containing \verb+make+ configuration files.
\item \verb+src+ C++ and Fortran source code of \emph{SW4}.
\item \verb+tools+ Matlab/Octave scripts for post processing and analysis.
\item \verb+pytest+ Python script and input files for testing the \emph{SW4} installation.
\item \verb+examples+ Sample input files.
\item \verb+Makefile+ Main makefile (don't change this file!).
\item \verb+CMakeLists.txt+ CMake configuration file (don't change this file either!).
\item \verb+wave.txt+ Text for printing the "SW4 Lives" banner at the end of a successful build.
\end{itemize}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Installing \emph{SW4} with make}\label{cha:installing-sw4}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%The \emph{SW4} source code is released under the GNU general public license and can be downloaded
%from:
%\begin{verbatim}
%https://computation.llnl.gov/casc/serpentine/software.html
%\end{verbatim}

The classical way of building \emph{SW4} uses \verb+make+. We recommend using GNU make, sometimes called
\verb+gmake+. You can check the version of make on you system with the command
\begin{verbatim}
shell> make -v
\end{verbatim}
If you don't have GNU make installed on your system, you can obtain it from www.gnu.org.

We have built \emph{SW4} and its supporting libraries on Intel based laptops and desktops running
LINUX and OSX. It has also been built on several supercomputers such as the Intel machines {\tt cab,
quartz}
(at LLNL) and {\tt edison, cori} (at LBNL), as well as the IBM BGQ machine {\tt vulcan} at LLNL. We have
successfully used the following versions of Gnu, Intel, and IBM compilers:
\begin{verbatim}
Gnu:   g++/gcc/gfortran      versions 4.5 to 7.2
Intel: icpc/icc/ifort        versions 16.0 to 18.0
IBM Blue Gene: xlcxx/xlc/xlf versions 12.1 to 14.1
\end{verbatim}

\emph{SW4} uses the message passing interface (MPI) standard (MPI-2 to be specific) for
communication on parallel distributed memory machines. Note that the MPI library often includes
wrappers for compiling, linking, and running of MPI programs. For example, the {\tt mpich2} package
build wrappers for the underlying C++ and Fortran compilers called {\tt mpicxx} and {\tt mpif90}, as
well as the {\tt mpirun} script. We highly recommend using these programs for compiling, linking,
and running \emph{SW4}.


\subsection{Basic compilation and linking of \emph{SW4}}\label{sec:basic-install}
\index{installation!basic} 

The best way of getting started is to first build \emph{SW4} without the proj.4 and cencalvm
libraries. This process should be very straight forward and the resulting \emph{SW4} executable
will support all commands except \verb+rfile+, \verb+efile+ and the \verb+proj+/\verb+ellps+ options in the
\verb+grid+ command. If you need to use these options, you can always recompile \emph{SW4} after the
proj.4 and cencalvm libraries have been installed. See \S~\ref{sec:cencalvm-install} for details.

The basic build process is controlled by the environmental variables \verb+FC+, \verb+CXX+,
\verb+EXTRA_FORT_FLAGS+, \verb+EXTRA_CXX_FLAGS+, and \verb+EXTRA_LINK_FLAGS+. These variables should
hold the names of the Fortan and C++ compilers, and any extra options that should be passed to
the compilers and linker. The easiest way of assigning these variables is by creating a file in the
\verb+configs+ directory called \verb+make.inc+. The \verb+Makefile+ will look for this file and
read it if it is available. There are several examples in the \verb+configs+ directory, e.g.
\verb+make.osx+ for Macs and \verb+make.linux+ for Linux machines. You should copy one of these files
to your own \verb+make.inc+ and edit it as needed. 

\subsubsection{Mac machines}
If you are on a Mac, you could copy the setup from \verb+make.osx+,
\begin{verbatim}
shell> cd configs
shell> cp make.osx make.inc
shell> cat make.inc
etree = no
proj = no
FC = mpif90
CXX = mpicxx
EXTRA_FORT_FLAGS = 
EXTRA_LINK_FLAGS = -framework Accelerate -L/opt/local/lib/gcc72 -lgfortran
\end{verbatim}
In this case, the \verb+blas+ and \verb+lapack+ libraries are assumed to be provided by the
\verb+-framework Accelerate+ option. The \verb+libgfortran+ library is located in the directory
\verb+/opt/local/lib/gcc72+, which is where \verb+macports+ currently installs it.

\subsubsection{Linux machines}
If you are on a Linux machine, we suggest you copy the configuration options from \verb+make.linux+,
\begin{verbatim}
shell> cd configs
shell> cp make.linux make.inc
shell> cat make.inc
FC = gfortran
CXX = mpicxx
EXTRA_LINK_FLAGS = -L/usr/lib64 -llapack -lblas -lgfortran
\end{verbatim}
This setup assumes that the \verb+blas+ and \verb+lapack+ libraries are located in the
\verb+/usr/lib64+ directory. 
In the case of Fedora Core 18, we needed to set the link flag variable to
\begin{verbatim}
EXTRA_LINK_FLAGS = -Wl,-rpath=/usr/lib64/mpich2/lib  -llapack -lblas -lgfortran
\end{verbatim}

\subsubsection{Using make}
You build \emph{SW4} with the "make" command from the main directory.
\begin{verbatim}
shell> cd /enter/your/path/sw4-v2.0
shell> make
\end{verbatim}
If all goes well, you will see the SW4 Lives banner on your screen after the compilation and linking
has completed,
{\samepage
\begin{verbatim}
    
``'-.,_,.-'``'-.,_,.='``'-.,_,.-'``'-.,_,.='````'-.,_,.-'``'-.,_,.='``


  _________    ____      __      ____    _    __
 /   ____  \   \   \    /  \    /   /   / |  |  |  
 |  |    \./    \   \  /    \  /   /   /  |  |  |
 |  |______      \   \/      \/   /   /   '--'  |
 \______   \      \              /    |______   |
        |  |       \     /\     /            |  |      
 /`\____|  |        \   /  \   /             |  |      
 \_________/         \_/    \_/              |__|      
                                       
   __       __  ____    ____  _______    ______    __  
  |  |     |  | \   \  /   / |   ____|  /    __|  |  | 
  |  |     |  |  \   \/   /  |  |__     |   (__   |  | 
  |  |     |  |   \      /   |   __|    \__    |  |  | 
  |  `----.|  |    \    /    |  |____    __)   |  |__| 
  |_______||__|     \__/     |_______|  (_____/   (__)
     

``'-.,_,.-'``'-.,_,.='``'-.,_,.-'``'-.,_,.='````'-.,_,.-'``'-.,_,.='``

\end{verbatim}
}
By default, \verb+make+ builds an optimized \verb+sw4+ executable. It is located in 
\begin{verbatim}
/enter/your/path/sw4-v2.0/optimize/sw4
\end{verbatim}
You can also build an executable with debugging symbols by adding the \verb+debug=yes+ option to \verb+make+,
\begin{verbatim}
shell> cd /enter/your/path/sw4-v2.0
shell> make debug=yes
\end{verbatim}
In this case, the executable will be located in
\begin{verbatim}
/enter/your/path/sw4-v2.0/debug/sw4
\end{verbatim}
It can be convenient to add the corresponding directory to your \verb+PATH+ environment
variable. This can be accomplished by modifying your shell configuration file, e.g. \verb+~/.cshrc+
if you are using C-shell.

\subsubsection{How do I setup the {\tt make.inc} file?}

The input file for \verb+make+ is
\begin{verbatim}
sw4-v2.0/Makefile
\end{verbatim}
Do {\em not} change this \verb+Makefile+. It should only be necessary to edit your configuration
file, that is,
\begin{verbatim}
/my/path/sw4-v2.0/configs/make.inc
\end{verbatim}
Note that you must create this file, for example by copying one of the \verb+make.xyz+ files in the
same directory. The \verb+make.inc+ file holds all information that is particular for your system,
such as the name of the compilers, the location of the third party libraries, and any extra
arguments that should be passed to the compiler or linker. This file also tells \verb+make+ whether
or not the \verb+cencalvm+ and \verb+proj.4+ libraries are available and where they are located.

The following \verb+make.inc+ file includes all configurable options:
\begin{verbatim}
etree = no
proj = no
SW4ROOT = /Users/petersson1

CXX = mpicxx
FC  = mpif77

EXTRA_CXX_FLAGS = -DUSING_MPI
EXTRA_FORT_FLAGS = -fno-underscoring
EXTRA_LINK_FLAGS = -framework vecLib
\end{verbatim}
The \verb+etree+ variable should be set to \verb+yes+ or \verb+no+, to indicate whether or not the
cencalvm and related libraries are available. The \verb+SW4ROOT+ variable is only used when
\verb+etree=yes+. The \verb+CXX+ and \verb+FC+ variables should be set to the names of the C++ and
Fortran compilers, respectively. Finally, the \verb+EXTRA_CXX_FLAGS+, \verb+EXTRA_FORT_FLAGS+, and
\verb+EXTRA_LINK_FLAGS+ variables should contain any additional arguments that need to be passed to
the C++ compiler, Fortran compiler, or linker, on your system.

\subsection{Building \emph{SW4} with proj.4 and/or efile support}
The installation of the proj.4, euclid, and cencalvm libraries is discussed in
Section~\ref{sec:cencalvm-install}. Note that the proj.4 libraray enables the more advanced
geographical mapping keywords in the {\tt grid} command and is also required by the {\tt rfile}
command. To enable the {\tt efile} command, you have to also install the {\tt euclid} and {\tt
  cencalvm} libraries. Note that the latter two libraries are only needed by the {\tt efile}
command. If you are not planning on using that command, there is no need to install those
libraies. This is a change from \emph{SW4} version 1.0.

Once you have successfully installed the proj.4, and optionally the euclid and cencalvm libraries,
it should be easy to re-configure \emph{SW4} to use them. Simply edit your configuration file
(\verb+make.inc+) by adding two lines to the top of the file, setting the {\tt etree} keyword to
{\tt yes} or {\tt no}, as appropriate.
\begin{verbatim}
proj = yes
etree = no
SW4ROOT = /thid/party/basedir
...
\end{verbatim}
You then need to re-compile \emph{SW4}. Go to the \emph{SW4} main directory, clean out the previous
object files and executable, and re-run make:
\begin{verbatim}
shell> cd /my/installation/dir/sw4-v2.0
shell> make clean
shell> make
\end{verbatim}
If all goes well, the ``SW4 lives'' banner is shown after the make command is
completed. As before, the \verb+sw4+ executable will be located in the \verb+optimize+ or
\verb+debug+ directories.

\subsection{Testing the \emph{SW4} installation}
The \emph{SW4} source code distribution includes a python(3) script for running several tests and
checking the solutions against previously verified results. Note that the same set of tests can be
performed when \emph{SW4} is built with CMake, see Section~\ref{cha:ctest-sw4}.

After \emph{SW4} has been built with \verb+make+, go to the \verb+pytest+ directory and run
\verb+test_sw4.py+. If the \verb+sw4+ executable resides in the \verb+optimize+ directory, you can
run the basic tests by doing:
\begin{verbatim}
shell> cd pytest
shell> ./test_sw4.py
\end{verbatim}
If all goes well, you should see the following output:
\begin{verbatim}
>shell test_sw4.py
Running all tests for level 0 ...
Starting test # 1 in directory: meshrefine with input file: refine-el-1.in
Test # 1 Input file: refine-el-1.in PASSED
Starting test # 2 in directory: meshrefine with input file: refine-att-1.in
Test # 2 Input file: refine-att-1.in PASSED
...
Starting test # 12 in directory: lamb with input file: lamb-1.in
Test # 12 Input file: lamb-1.in PASSED
Out of 12 tests, 12 passed and 0 failed.
\end{verbatim}
Some aspects of the testing can be modified by providing command line arguments to
\verb+test_sw4.py+. For a complete list of options do \verb+test_sw4.py --help+, which currently
give the output:
\begin{verbatim}
shell> ./test_sw4.py --help
usage: test_sw4.py [-h] [-v] [-l {0,1,2}] [-m MPITASKS] [-d SW4_EXE_DIR]

optional arguments:
  -h, --help            show this help message and exit
  -v, --verbose         increase output verbosity
  -l {0,1,2}, --level {0,1,2}
                        testing level
  -m MPITASKS, --mpitasks MPITASKS
                        number of mpi tasks
  -d SW4_EXE_DIR, --sw4_exe_dir SW4_EXE_DIR
                        name of directory for sw4 executable
\end{verbatim}
Note  that the directory name for the \verb+sw4+ executable should be given relative to the main
\verb+sw4+ directory.

\section{Installing \emph{SW4} with CMake}\label{cha:installing-cmake-sw4}
\emph{SW4} can also be built with CMake. Compared to using regular {\tt make}, this build process is
easier to use because it is fully automated. However, it gives the user less control of which
compilers, linker, and libraries to use. Similar to using regular {\tt make}, the \emph{SW4} CMake
configuration allows automated correctness testing of the installation. The test runs the same set
of cases as the \verb+test_sw4.py+ script in the \verb+pytest+ directory, see
Section~\ref{cha:ctest-sw4} for details.

To use CMake, navigate to the top {\tt sw4} directory and run the following commands:
\begin{verbatim}
shell> mkdir build
shell> cd build
shell> cmake [options] ..
shell> make
\end{verbatim}
The two dots after {\tt cmake [options]} are essential and instructs it to look in the parent
directory for the {\tt CMakeLists.txt} file.
 
The \verb+cmake+ command searches for the necessary libraries and other dependencies, then creates
makefiles that are appropriate for your system. You then run \verb+make+ to compiles and link
\emph{SW4} using these makefiles. For details about the exact commands being used in compilation,
run \texttt{make VERBOSE=1}.  Once SW4 has been successfully built, you will see the
``SW4 Lives!'' banner on the screen.

NOTE: \verb+cmake+ puts the \verb+sw4+ executable in the \verb+build/bin+ directory. This is
different from regular {\tt make}, which puts the executable in the \verb+optimize+
directory.

NOTE: If you want to rebuild \verb+sw4+ with a new set of options, you can force \verb+cmake+ to start
from scratch by removing the file \verb+CMakeCache.txt+ in the \verb+build+ directory. Another way
is to remove all files in the \verb+build+ directory.

\subsection{CMake Options}
CMake provides several options to allow customized configuration of \emph{SW4}.  To use any option,
add \texttt{-D\textless option\textgreater=\textless value\textgreater} to the options in the
\texttt{cmake} command.  For example:
%
\begin{verbatim}
cmake -DTESTING_LEVEL=1 -DCMAKE_BUILD_TYPE=Debug ..
\end{verbatim}
%
configures \emph{SW4} with testing level 1, to be compiled with debugging symbols in the object
files. A list of options is shown in the table below.
%
\begin{center}
\begin{tabular}{|l|c|p{0.55\textwidth}|}
\hline
Option & Default & Details \\
\hline
%
PROJ4\_HOME & (none) & The path to the Proj.4 installation to use when compiling
\emph{SW4}. \\ \hline
%
CENCALVM\_HOME & (none) & The path to the cencalvm installation to use when compiling
\emph{SW4}. \\ \hline
%
CMAKE\_BUILD\_TYPE & Release & The type of build to setup. Can be either \texttt{Debug},
\texttt{Release}, or \texttt{RelWithDebInfo}.  This affects the type of optimization and debug flags
used in compiling \emph{SW4}. \\ \hline
%
TESTING\_LEVEL & 0 & Specifies the testing level for automated tests.  Level 0 corresponds to tests
that run in roughly a minute or less (7 total), level 1 to tests that run in roughly 10 minutes or
less (13 total) and level 2 to tests that may require up to an hour or more (17 total). \\ \hline
%
MPI\_NUM\_TEST\_PROCS & 4 & Number of MPI processes to use in tests. Generally using more processes
will result in the tests finishing faster, but there is no point exceeding the number of available
cores on your system.  We strongly recommend at least 8 processes if TESTING\_LEVEL is 1 or
higher.\\ \hline
%
% -DMPIEXEC=/usr/bin/srun
%
MPIEXEC & mpirun & UNIX command for running an MPI application.\\ \hline
% -DMPIEXEC_NUMPROC_FLAG=-n
MPIEXEC\_NUMPROC\_FLAG & -np & MPI command option for specifying the number of processes.\\ \hline 
% -DMPIEXEC_PREFLAGS=-ppdebug
MPIEXEC\_PREFLAGS & (none) & Extra MPI command option.\\ \hline 
\end{tabular}
\end{center}

\paragraph{Modifying the MPI execution commands.}
By default, \verb+mpirun+ is used to start parallel runs when you do \verb+make test+.  However, on
Livermore computing (LC) machines the command for running MPI programs is \verb+srun+, not
\verb+mpirun+. Also, the flag for specifying the number of processors is different, and you must
give an additional flag for running interactive jobs on the debug partition. For example, you would
say
\begin{verbatim}
srun -ppdebug -n 128 sw4 inputfile.in
\end{verbatim}
to run on the debug partition using 128 cores. To modify the default MPI execution program and
other runtime parameters, the variables MPIEXEC, MPIEXEC\_NUMPROC\_FLAG, and
MPIEXEC\_PREFLAGS can be set as in the following example:
\begin{verbatim}
cmake -DTESTING_LEVEL=2 -DMPI_NUM_TEST_PROCS=128 -DMPIEXEC=/usr/bin/srun \
      -DMPIEXEC_NUMPROC_FLAG=-n -DMPIEXEC_PREFLAGS=-ppdebug ..
\end{verbatim}

After the proj.4, euclid and cencalvm libraries have been installed (see next section), you need to tell
\verb+cmake+ where to find them. On the LC-machines, all three libraries are currently installed under
\verb+/usr/apps/wpp+, and you can use the following command options to configure \verb+sw4+:
\begin{verbatim}
cmake -DTESTING_LEVEL=2 -DMPI_NUM_TEST_PROCS=36 -DMPIEXEC=/usr/bin/srun \
      -DMPIEXEC_NUMPROC_FLAG=-n -DMPIEXEC_PREFLAGS=-ppdebug \
      -DPROJ4_HOME=/usr/apps/wpp -DCENCALVM_HOME=/usr/apps/wpp ..
\end{verbatim}
To verify that \verb+cmake+ actually found the libraries, pay attention to the
following lines of the output from the \verb+cmake+ command:
\begin{verbatim}
...
-- Found PROJ4: /usr/apps/wpp/lib/libproj.so  
-- Found CENCALVM: /usr/apps/wpp/lib/libcencalvm.so;/usr/apps/wpp/lib/libetree.so  
...
\end{verbatim}

Sometimes CMake doesn't pick up the correct compiler. Say, for example that the C++ compiler on your
system is called {\tt mpicxx} and the Fortran compiler is {\tt mpiifort}. You can tell {\tt cmake}
to use those compilers by setting the following envoronment variables {\em before} running cmake
(assuming a {\tt csh} shell),
\begin{verbatim}
  > setenv CXX mpicxx
  > setenv FC mpiifort
\end{verbatim}

\subsection{CTest}\label{cha:ctest-sw4}
The \emph{SW4} CMake configuration includes several test cases that are used to verify the correctness
of the \emph{SW4} installation.  Each test consists of two parts. First it runs a case using an input
file in the \verb+pytest+ directory. Secondly, it checks that the results are within
a reasonable error tolerance from previously recorded results.

To run the tests, use either the command \texttt{make test} or \texttt{ctest} as follows:
%
\begin{verbatim}
build > ctest
Test project /Users/petersson1/src/sw4-cig/build
      Start  1: Run_twilight/flat-twi-1
 1/24 Test  #1: Run_twilight/flat-twi-1 .....................   Passed    0.49 sec
      Start  2: Check_Result_twilight/flat-twi-1
 2/24 Test  #2: Check_Result_twilight/flat-twi-1 ............   Passed    0.03 sec
      Start  3: Run_twilight/flat-twi-2
...
      Start 23: Run_pointsource/pointsource-sg-1
23/24 Test #23: Run_pointsource/pointsource-sg-1 ............   Passed   89.56 sec
      Start 24: Check_Result_pointsource/pointsource-sg-1
24/24 Test #24: Check_Result_pointsource/pointsource-sg-1 ...   Passed    0.03 sec

100\% tests passed, 0 tests failed out of 24

Total Test time (real) = 230.91 sec
\end{verbatim}

You can run tests selectively using \texttt{ctest -R \textless regex\textgreater}, for example:
%
\begin{verbatim}
build > ctest -R meshrefine
Test project /Users/petersson1/src/sw4-cig/build
    Start 15: Run_meshrefine/refine-el-1
1/6 Test #15: Run_meshrefine/refine-el-1 .................   Passed   25.61 sec
    Start 16: Check_Result_meshrefine/refine-el-1
2/6 Test #16: Check_Result_meshrefine/refine-el-1 ........   Passed    0.03 sec
    Start 17: Run_meshrefine/refine-att-1
3/6 Test #17: Run_meshrefine/refine-att-1 ................   Passed   22.00 sec
    Start 18: Check_Result_meshrefine/refine-att-1
4/6 Test #18: Check_Result_meshrefine/refine-att-1 .......   Passed    0.03 sec
    Start 19: Run_meshrefine/refine-att-2nd-1
5/6 Test #19: Run_meshrefine/refine-att-2nd-1 ............   Passed   17.63 sec
    Start 20: Check_Result_meshrefine/refine-att-2nd-1
6/6 Test #20: Check_Result_meshrefine/refine-att-2nd-1 ...   Passed    0.03 sec

100% tests passed, 0 tests failed out of 6

Total Test time (real) =  65.35 sec
\end{verbatim}

If a test fails you can check the details in the output log at
\texttt{Testing/Temporary/LastTest.log}.

\section{Installing the proj.4, euclid, and cencalvm packages}\label{sec:cencalvm-install}
\index{installation!cencalvm} \index{installation!efile}

If you are only interested in using the advanced geographical mapping options of the {\tt grid} command,
or the {\tt rfile} command, you only need to install the proj.4 package.

The following instructions describe how to install all three packages. For simplicity all packages
are installed under the same top directory. If you are using {\tt cmake}, you may optionally put the
proj.4 package in a separate directory. In the following we shall assume that all packages are
installed under the same top directory, and that you assign the name of that directory to the
environment variable \verb+SW4ROOT+. When you are finished installing the packages, the
corresponding include and library files should be in the sub-directories \verb+${SW4ROOT}/include+
and \verb+${SW4ROOT}/lib+, respectively.

The cencalvm library was developed by Brad Aagaard at USGS. Instructions for building the cencalvm
library as well as downloading the Etree database files for Northern California, can
currently be downloaded from
\begin{verbatim}
http://earthquake.usgs.gov/data/3dgeologic/cencalvm_doc/INSTALL.html
\end{verbatim}
The installation process for cencalvm, which is outlined below, is described in detail on the above
web page.  Note that cencalvm relies on both the euclid and the proj.4 libraries.

The euclid library must be installed manually by explicitly copying all include files to
the include directory and all libraries to the lib directory,
\begin{verbatim}
shell> cd euclid3-1.2/libsrc
shell> make
shell> cp *.h ${SW4ROOT}/include
shell> cp libetree.* ${SW4ROOT}/lib
\end{verbatim}
The proj4 library should be configured to be installed in \verb+${SW4ROOT}+. This is accomplished by
\begin{verbatim}
shell> cd proj-4.7.0
shell> configure --prefix=${SW4ROOT}
shell> make
shell> make install
\end{verbatim}
We remark that the {\tt proj.4} library can alternatively be installed using {\tt macports} (if you are
on a Mac OSX machine).

The cencalvm library should also be configured to be installed in \verb+${SW4ROOT}+. You also have to help
the configure script finding the include and library files for the proj.4 and etree libraries,
\begin{verbatim}
shell> cd cencalvm-0.6.5
shell> configure --prefix=${SW4ROOT} CPPFLAGS="-I${SW4ROOT}/include" \
                 LDFLAGS="-L${SW4ROOT}/lib"
shell> make
shell> make install
\end{verbatim}

To verify that the libraries have been installed properly, you should go to the \verb+SW4ROOT+
directory and list the {\tt lib} sub-directory (\verb+cd ${SW4ROOT}; ls lib+). You should see the
following files (on Mac OSX machines, the .so extension is replaced by .dylib ):
\begin{verbatim}
shell> cd ${SW4ROOT}
shell> ls lib
libetree.so libetree.a 
libproj.so libproj.a libproj.la 
libcencalvm.a libcencalvm.la libcencalvm.so 
\end{verbatim}
Furthermore, if you list the include sub-directory, you should see include files such as 
\begin{verbatim}
shell> cd ${SW4ROOT} %$
shell> ls include
btree.h etree.h etree_inttypes.h
nad_list.h projects.h proj_api.h
cencalvm
\end{verbatim}
Note that the include files for cencalvm are in the sub-directory with the same name.

\section{Disclaimer} 
This document was prepared as an account of work sponsored by an agency of the United States
government. Neither the United States government nor Lawrence Livermore National Security, LLC, nor
any of their employees makes any warranty, expressed or implied, or assumes any legal liability or
responsibility for the accuracy, completeness, or usefulness of any information, apparatus, product,
or process disclosed, or represents that its use would not infringe privately owned
rights. Reference herein to any specific commercial product, process, or service by trade name,
trademark, manufacturer, or otherwise does not necessarily constitute or imply its endorsement,
recommendation, or favoring by the United States government or Lawrence Livermore National Security,
LLC. The views and opinions of authors expressed herein do not necessarily state or reflect those of
the United States government or Lawrence Livermore National Security, LLC, and shall not be used for
advertising or product endorsement purposes. 

\bibliographystyle{plain}

\bibliography{sw4-refs} 

\end{document}
